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ABSTRACT 

. report describes an external review of education reform 

in Oklahoma that focuses on standards, assessment, and accountability 
Analysis of written documents and interviews with leaders from government 
education, business, and other stakeholders indicate that the biggest 
strength of Oklahoma's education system is the widely shared, genuine 
comnitment to improving opportunities for all young people. Results find that 
Oklahoma s standards and assessments and ACT ' s Standards for Transition have 
significant strengths, though there are also shortcomings. The assessment 
system provides measures of student progress against state standards, as well 
as measures permitting comparisons with students in other states; includes 
tests in a broad number of subjects; and uses the ACT to good advantage 
However, the array of tests is disjointed. Strengths of the accountability 
system include: reports on performance are widely disseminated, high stakes 
appear to motivate students and schools to succeed, and higher education is a 
genuine partner in school accountability. Some of the many problematic areas 
include: it is unclear how useful test results are to schools and the public 
interventions for low-performing schools must be strengthened, and the system 
has few stakes for anyone. Recommendations for improvement in each area are 
included. Three appendices present review team biographies, references, and 
interviews. (SM) 
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Achieve, Inc, 



Achieve is an independent, bipartisan, nonprofit organization created by governors and 
corporate leaders to help states and the private sector raise standards and performance in 
America’s schools. Founded at the 1996 National Education Summit, Achieve has spon- 
sored two additional Summits in 1999 and 2001. 



Achieve helps states raise academic standards, measure performance against those standards, 
establish clear accountability for results'and strengthen pubHc confidence in our education 
system. To do this, we: 



D help states benchmark their standards, assessments and accountability systems against 
the best in the country and the world; 

□ provide sustained public leadership and advocacy for the movement to raise stan- 
dards and improve student performance; 

□ build partnerships that allow states to work together to improve teaching and learning 
and raise student achievement; and 

□ serve as a national clearinghouse on education standards and school reform. 
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August 2002 



About This Review 



In summer 2001, in response to a request from the Oklahoma Business and Education Coalition 
(OBEC), the Oklahoma State Regents for Higher Education, the Oklahoma State Department of 
Education, and the Governor’s Office, Achieve organized an external review of the state’s efforts 
to establish a standards-based system of education. The review evolved to be a two-part study. 
The first was a policy-level analysis of the larger issues and decisions that govern the progress of 
reform in the state, specifically its success in establishing a system of standards, assessments and 
accountability. The second was a detailed study of the strengths and weaknesses of Oklahoma’s 
current system of standards and assessments in English language arts and mathematics, including 
the related standards and assessments of the American College Testing program (ACT) with 
which Oklahoma has formed a close partnership. 

To carry out this policy review. Achieve assembled a team with expertise in the areas that were 
of particular concern to Oklahoma reform leaders (brief biographies of the review team members 
are included in Appendix A). The review team analyzed a comprehensive set of written 
documents dealing with key aspects of Oklahoma’s education system, with a special emphasis on 
standards, assessments and accountability (a list of these documents is provided in Appendix B). 
Achieve team members visited Oklahoma for two days in November 2001, interviewing a cross- 
section of leaders from government, business, education and other stakeholder groups, as well as 
senior staff from the Oklahoma State Department of Education (a list of those interviewed is 
contained in Appendix C). Achieve reviewers compared their reactions and impressions and 
commented on major findings. Although Achieve invited the executive committee of OBEC to 
review a draft for factual accuracy, the observations and conclusions are entirely our own. 

We are keenly aware of the limits of this kind of review and of the risks inherent in offering up 
findings and recommendations based upon a limited exposure to a complex set of issues 
embedded in the state’s unique history and traditions. We are equally aware that, despite the real 
progress we have made over the past decade in learning what works in education reform, there is 
much that we do not yet know. Nonetheless, in this report we have strived to provide state 
policymakers with our best judgment as to the current status of education reform in Oklahoma 
and areas for improvement over the next decade. We appreciate the invitation of Oklahoma 
leaders to open their system to review by Achieve and hope our report will prove helpful to those 
with policymaking responsibility for the education of Oklahoma’s students. 

Achieve has conducted similar reviews of other state systems, most recently at the request of 
education policymakers in Maryland and Texas. Those policy reviews can be read on Achieve’s 
Web site, www.achieve.org. 
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About Achieve, Inc. 



Achieve, Inc., is an independent, nonprofit, bipartisan organization created after the 1996 
National Education Summit by the nation’s governors and business leaders to help states raise 
academic standards and improve schools. Achieve provides advice and assistance to state policy 
leaders on issues of academic standards, assessments and accountability. It has a small staff, 
augmented by a team of senior advisers, and conducts much of its work in partnership with other 
education and business organizations. Under the auspices of Achieve’ s Benchmarking Initiative, 
16 states have sought Achieve’s external reviews of state education policy issues since 1998. 

To carry out this review. Achieve drew upon several nationally respected experts: Ruben 
Carriedo, senior research associate at the University of Michigan School of Education; Paul 
Reville, executive director of the Massachusetts Business Alliance for Education and the Pew 
Forum on Standards-Based Reform; Janis Somerville, senior associate with the University 
System of Maryland and the National Association of System Heads for colleges and universities; 
and John Stevens, executive director of the Texas Business and Education Coalition. Matthew 
Gandal, Achieve’s executive vice president, chaired the review team. Robert Rothman, formerly 
a senior project associate at Achieve, and Jean Slattery, director of Achieve’s Benchmarking 
Initiative, also participated in the review. Slattery organized the review and co-authored this 
report with Rothman. 
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Executive Summary, Part I 



In 2001, the Oklahoma Business and Education Coalition (OB EC), the Oklahoma State Regents 
for Higher Education, the Oklahoma State Department of Education, and the Governor’s Office, 
asked Achieve, Inc., to undertake an independent review of the state’s policies and practices to 
improve schools by setting high academic standards and holding schools and students 
accountable for results. 

In examining the state’s record of reform and talking with a wide cross-section of residents, it 
was clear to Achieve ’s team of experts that standards, assessments and accountability have been 
at the heart of Oklahoma’s efforts to improve its schools for more than a decade. Landmark 
legislation in 1990 set the state on a course to create content standards and begin statewide 
testing five years later. Along the way, the state has sent a clear signal about its commitment to 
reform, enacting standards and tests and initiating a limited accountability system despite its 
fiscal constraints. Both higher education and the private sector have been significant contributors 
to the state’s success. 

This is important context as Oklahoma enters the next phase of standards-based reform. The state 
is preparing for a periodic review of its standards, which provides an ideal opportunity to 
strengthen them and the tests aligned to them at a time when accountability based on academic 
results should increase. As is the case with the other 49 states, Oklahoma will move forward in 
the context of the No Child Left Behind Act, the new federal education law whose provisions 
include expansion of state testing, attention to the achievement of all students and shared 
accountability for results. 

While Achieve does not minimize the work that must be done or the challenges posed by doing it 
in a large rural state with a diverse population, we consider Oklahoma well positioned to make 
significant progress. As state government, K-12 and higher education, and the private sector 
work together to make progress in Oklahoma’s next phase of reform, we urge policymakers and 
others interested in supporting school improvement to focus their efforts on a few key goals: 

Strengthen the Priority Academic Student Skills (PASS) so that standards provide a 
more challenging foundation for school improvement efforts by the state and local 
school districts. 

In addition to this review of its education refprm policies, Oklahoma asked Achieve to conduct a 
comprehensive study of its standards and tests in English and mathematics and their relationship 
to ACT, which the state has used to good effect to communicate expectations for college 
preparedness and help more students meet them. Measuring Up, the resulting report that serves 
as a companion to this policy review, offers a road map to improving the state’s standards and 
tests. Oklahoma’s PASS has some strong features on which to build. They are comprehensive, 
measurable, generally compatible with the ACT’s Standards for Transition and 
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tests, and, in some areas, quite rigorous. In particular, Oklahoma’s math standards lay strong 
conceptual foundations for grades K-5 and, overall, contain a well-developed sequence of 
knowledge and skills. 

However, Achieve noted significant areas for improvement, which Oklahoma should strive to 
address through the upcoming, legislatively mandated review of PASS. These include clarifying 
the level of rigor expected by the standards through use of suggested reading lists or sample text 
in English language arts and sample problems in math and restructuring the English language 
arts standards so that the development of skills from grade to grade is explicit and the most 
important content receives the most emphasis. 

Leverage grade-by-grade testing provisions of the new federal education law to create a 

coherent testing system that promotes challenging expectations for all students. 

Oklahoma has made some good choices in constructing its assessment system, such as directly 
measuring students’ ability to compose written prose and testing in subjects beyond reading and 
math. 

While the state built a strong foundation with its assessment system. Achieve identified four 
major challenges with which policymakers need to contend. First, while each test offers results 
that may be useful in some settings, the mix of state standards-based, national norm-referenced 
and college admissions tests may be sending a mixed signal about the primacy of the content 
found in Oklahoma’s standards. Second, the level of rigor fluctuates from test to test. Third, 
Oklahoma lacks English and math tests in grades 4, 6 and 7 required by the new federal law. 
Fourth, at the high school level, the state needs to add to its battery of tests to cover coursework 
students pursue after 10* grade. Oklahoma’s goal should be a set of tests from elementary to 
high school that are tightly aligned with PASS and can provide consistent and comparable results 
to educators and parents. The State Department of Education is laying the groundwork for such a 
coherent testing program and plans to take the worthwhile step of engaging outside experts to 
ensure the tests are vertically aligned from one grade to the next, creating an appropriate 
progression of skills and knowledge students are expected to master. 

Enhance the accountability system to sharpen its focus on the achievement of all 

students and to create incentives for improvement among more schools, teachers and 

students. 

Oklahoma has seen results improve when consequences are attached to performance; the higher 
test scores under the Reading Sufficiency Act and a 10-percentage point percent jump in the 
passing rate on the 8*-grade reading test after a passing score was required to obtain a driver’s 
license have shown that accountability works. But beyond 8* graders and elementary school 
educators, there are no significant consequences for performance — good or bad — for others in 
the education system. The state only identifies the lowest-performing schools, and the 
expectations for acceptable performance are so low that few schools earn the rating for more 
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than a year at a time. As a result, the accountability system does not lead to effective targeting of 
assistance and resources to the schools most in rieed. In particular, minority students may be at 
risk because the state historically did not produce achievement data that drew attention to their 
lagging performance. We encourage Oklahomans to strengthen the state’s commitment to 
helping all children achieve by bolstering the accountability system. A vital step would be the 
creation of a data system to track students’ performance from grade to grade and place to place. 
Once it identifies schools as low performing, the state must ensure they get adequate and 
effective assistance. If schools fail to improve after receiving that assistance, more dramatic 
intervention should occur. The state should be better positioned to take these steps now that test 
results are reported in disaggregated form and an Academic Performance Index has been added 
to create incentives for all schools to improve. 

Report achievement results clearly so that they are more useful to schools and to the 

public. 

Oklahoma has made a commitment to report fully on school and student performance. While the 
supply of information appears to be ample, its delivery may not be as effective as it could be. 
Achieve’ s review team heard differing opinions about the usefulness of performance reports to 
schools in planning for improvement. Public accountability could be enhanced by replacing the 
separate reports produced by the State Department of Education and the Governor’s Office with 
a single, jointly published report, based on a common set of performance indicators and other 
data. It is needlessly confusing for the public to have to sift and sort related information from two 
different sources. 

Tie achievement of the state’s standards more closely to opportunities to succeed after 

high school, such as college admissions and employment. 

Oklahoma has taken important steps to tie its overall efforts to raise student achievement to its 
goal of increasing the number of students who attend college. The state has steadily increased the 
percentage of high school juniors and seniors taking the ACT and has seen its average scores rise 
at the same time. Not surprisingly given their different purposes and origins, the ACT and 
Oklahorna’s standards and tests are compatible, but not completely consistent. To increase the 
likelihood of students succeeding on both state tests and the ACT, Oklahoma should judiciously 
add key elements of the ACT to its standards and tests. In addition, to safeguard their academic 
rigor, the state’s career and technology (CareerTech) education programs should not be allowed 
to function apart from the accountability system for other schools and students. Responsibility 
for CareerTech programs serving high school students should lie with the State Superintendent 
of Public Instruction to ensure that they deliver solid academic preparation for the resources 
allotted and prepare their graduates to succeed in higher education and to meet current workplace 
demands. 
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Forge a statewide consensus in support of common, rigorous standards for all students. 

Oklahoma has maintained efforts to raise standards for more than a decade, initiating innovative 
programs such as the leveraging of the ACT exam to bring earlier attention to college 
preparation. The state faces a time of transition, in terms of both revising its standards and 
selecting its political leadership. Some in Oklahoma question whether the notion of raising all 
students to high standards is a universally accepted view. Balancing accountability with the 
autonomy of the state’s more than 500 local school districts remains a critical concern for many. 

In our view, traditions of local control are not at odds with the idea that all students should 
master a common body of knowledge and skills. Both state and local leaders bear the 
responsibility of ensuring that a high school diploma in Oklahoma actually signifies readiness to 
succeed in higher education or fulfilling careers. It will be critical for the business and 
postsecofidary education communities to raise their voices in support of this idea and to continue 
to make the connection between well-educated citizenry and the state’s economic and civic 
vitality. 
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Education Reform: The National Perspective 



Last October, governors, corporate leaders, state education leaders and educators gathered to 
shape the next phase of school improvement at the 2001 National Education Summit. The 
Summit meeting marked only the fourth time in American history that education policymakers 
have met to discuss common challenges and define common solutions — and the first time 
teachers and principals attended. These Summits have played a pivotal role in defining a 
consensus view to guide states’ school improvement efforts. 

In 1989, President George H. W. Bush and the 50 governors set broad, bold goals for the 
nation’s education system at every level from early childhood through adult training and 
development. In 1996, governors and business executives committed to establish clear, 
challenging expectations for what students should know and be able to do in elementary and 
secondary school; regular measurement of student and school performance; and public 
accountability for results. In 1999, governors, CEOs and education leaders focused on concrete 
actions needed to make these ideas a reality in classrooms: improving the quality of teaching, 
strengthening accountability and putting in place the supports needed to help all students achieve 
high standards. 

Participants at the 2001 Summit advanced three sets of principles to help boost student performance 
across the board while closing the achievement gap: improving state assessment systems to direct 
resources and support where they are needed most; developing firm, fair and balanced accountability 
systems that will guarantee all students an equal opportunity to achieve high standards; and creating 
and sustaining a top-flight education workforce while injecting responsibility for results into the 
profession. Educators and executives from government and business reaffirmed their commitment to 
the twin goals of excellence and equity in America’s schools: 

We must raise achievement for all students while closing the achievement gap 
separating the educational “haves” from the “have-nots.” These goals are an 
irreducible educational minimum for the United States. Nothing less than their 
full attainment will serve the nation’s social, democratic and economic interests. 

Many states have come a long way since the 1996 Summit. Virtually all states have put in place 
standards and tests to assess educational progress in the core academic areas, and nearly half are 
developing incentives and consequences for schools, districts and students tied to results. Yet, as 
standards-based reform enters its second decade, new challenges loom. No state has eliminated 
the achievement gap once and for all. Test results are just beginning to count for students and 
schools in most states. And while many students are learning more and test scores are gradually 
improving, more dramatic improvements are needed for all students to succeed. Public 
confidence in schools is rising, yet, at the same time, a small but vocal minority of reform critics 
are urging policymakers to turn back to the pre-standards era. Most recently, the law governing 
federal involvement in schools — the Elementary and Secondary Education Act — will ask 
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states to do more and do better when it comes to assessment and accountability. States will be 
held accountable to national policymakers for ensuring that schools make progress toward 
meeting standards, and new testing systems will be needed to gauge their annual progress. 
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Education Reform in Oklahoma 



Readiness for Reform 

In looking to improve and accelerate standards-based reform efforts, Oklahoma is building off an 
impressive foundation. The state embraced the standards agenda earlier than most, with 
comprehensive legislation adopted in 1990 that set in motion a series of policies that have guided 
state actions since. In the wake of the legislation, the state adopted content standards outlining 
what all students are expected to know and be able to do and has revised them continually. It 
developed and implemented tests to measure student attainment of the standards as early as 
1995. And the state has put in place a number of measures to hold students and schools 
accountable for performance. 

These efforts are particularly impressive given the circumstances under which state officials act. 
In contrast to many states with a long tradition of state-level education policy, Oklahoma is 
proud of local control. With more than 500 school districts, many of which are quite small, local 
districts represent strong power centers. In many cases, the school districts are the largest 
employers in their commuriities. 

Oklahoma also has moved to improve its education system despite severe fiscal constraints. The 
Achieve review team heard over and over during its visit that Oklahoma is a poor state with a 
large rural population that does not have a lot of money to spend on education. The reviewers 
also heard repeatedly that Oklahoma teachers are among the lowest-paid in the nation. Education 
reform is not without cost, and the fact that the state has been able to implement some significant 
changes despite its limited resources suggests that the priorities of its leaders are right. 

As the state moves forward, it can build on a number of substantial assets, in addition to its 
strong foundation in standards-based reform. One significant plus is the active and committed 
leadership of higher education. The strong involyement of colleges and universities in K-12 
reform is a rare and welcome sign. Hans Brisch, the chancellor of the Oklahoma State Regents 
for Higher Education, and the regents have taken bold steps to improve the preparation of 
Oklahoma students for higher education by teaming up with the ACT program to make the ACT 
series of tests widely used throughout the state and by strengthening admissions standards. These 
initiatives affect most Oklahoma students, and they appear to be widely embraced. 

Another significant asset is the strong interest and involvement of the business community. 
Business leaders from a variety of industries appear actively engaged and willing to work with 
elected leaders to improve education. The formation of the OBEC is a powerful sign that 
business leaders intend to remain engaged and are willing to invest resources in creating an 
institution to promote their involvement. At the same time, the success of the 2001 MAPS for 
Kids initiative in Oklahoma City, which called for extending a tax increase and issuing a bond 
for educational improvements, demonstrates that business leaders’ involvement can pay off. 



Aiming Higher - Oklahoma 



Achieve, Inc., 2002 



13 



ERIC 




Getting 60 percent of voters to approve an increase in taxes is rarely easy, but the partnership of 
business leaders and elected officials from both major political parties made it happen. 

Despite many positive signs, the Achieve review team also heard a number of questions that 
Oklahoma should answer as it moves forward in strengthening its standards, assessments and 
accountability systems. One question is whether the many actors involved in education reform 
speak with a unified voice. Achieve has found in other states that, while top leaders from various 
sectors may disagree on strategies, a common message about goals is essential. For one thing, it 
shows students, parents and schools that there is one set of expectations for performance. 

Sending mixed messages could be confusing and could complicate efforts to reach the common 
goals. 

In addition, a unified voice can help ensure continued support for the reforms. This is 
particularly important at a time of political transition, as Oklahoma currently faces. The common 
refrain can reassure the public that the agenda will move forward no matter who controls the 
legislature or the governor’s mansion. In Texas, for example, unified leadership from both 
political parties, business and education has helped ensure that that state’s reform effort remained 
intact over a decade and over three governorships (including a change from Democratic control 
to Republican). 

Another question the review team heard from Oklahomans was whether the entire education and 
political community shared a desire to raise standards and achievement. To be sure, the 
commitment from the Governor’s Office, the State Department of Education, the Regents for 
Higher Education and businesses is tangible and strong. But the review team also heard some 
reluctance from rural areas and from their elected representatives. There was a sense of 
satisfaction with the current system and a concern that changes could damage what they had and 
produce a generation no longer willing to stay in those rural areas. 

The review team also heard questions about the legislature’s practice of authorizing reforms 
without providing the money to carry them out. Although funding is always an issue, did these 
practices indicate that the legislature is lukewarm about the reforms? Why has there been so little 
outcry about this practice from supporters of the reforms? 

With these questions in mind. Achieve is pleased to offer its observations and recommendations 
about Oklahoma’s standards, assessment and accountability systems. As they chart a course for 
the next phase in the state’s decade-long reform effort, we hope state leaders will find Achieve’s 
perspective on standards, assessment and accountability to be of value. In offering these findings, 
we must add that these reform elements by themselves do not produce better schools and better- 
educated students. They provide information and incentives to enable students, parents, teachers, 
principals and public officials to make the changes needed to improve learning; we do not want 
to minimize the importance of those changes. Schools need teachers who can teach all students 
to reach high standards and principals who can lead their staffs to do so. Districts also need the 
capacity to assist schools in their efforts to improve teaching and learning. Although issues of 




Aiming Higher - Oklahoma 



14 



15 



Achieve, Inc., 2002 



capacity — specifically, teacher quality and the knowledge and skills of educators — are beyond 
the scope of this report, we must emphasize that no significant improvement will occur until the 
state addresses them, in addition to strengthening standards, assessments and accountability. 

A Propitious Time 

Oklahoma’s examination of its standards, assessment and accountability policies comes at a 
propitious time. The state is about to embark on a periodic review of its standards. As standards 
are revised, assessments must be upgraded to ensure that they remain aligned to the standards. 

As tests are revised, it makes sense to take a hard look at the totality of the assessment system to 
ensure that all tests are serving useful purposes. It is important, as well, to examine the uses to 
which the assessments are put to ensure that they provide appropriate information and incentives 
for students, educators and the public. 

Timing also is important because of the enactment of the federal No Child Left Behind Act, 
which will require all states to re-examine their assessment and accountability systems and make 
significant changes. Specifically, the law requires states to put in place annual reading and 
mathematics tests, aligned with state standards, in grades 3 through 8. Test results must be 
broken down by race, gender, income and other factors. The law also requires states to set a 
specific trajectory to allow all students to reach proficiency within 12 years. Schools that fail to 
demonstrate adequate progress toward that goal will be subject to a variety of sanctions. 

The statute has enormous implications for virtually all states, including Oklahoma. Currently 
only nine states meet the law’s requirements for annual tests in grades 3 through 8, and few have 
in place all the components of accountability the law requires. Complying with the law 
represents an opportunity to remake assessment and accountability systems so that they work 
more effectively. The challenge will be to do so in the right way and avoid easy steps that meet 
the letter of the law but fail to capture its spirit. The Achieve review clearly shows that many 
Oklahomans had been heading in the direction the No Child Left Behind Act points, even before 
the law’s outlines became clear. The statute thus provides rules that can guide the state as it 
pursues the reform course it already is on. 
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Major Findings: Standards and Assessment Benchmarking 



As noted earlier, in addition to requesting a review of its education reform policies, Oklahoma 
also requested that Achieve undertake a comprehensive study of its standards and assessihents in 
English language arts and mathematics — including examining die degree of fit between ACT’s 
Standards for Transition, ACT 1 1/12, tests and Oklahoma’s standards and assessments. While 
the results of this standards benchmarking and alignment study are explained in detail in Part II 
of Achieve’s report to Oklahoma, Measuring Up, a summary of key findings follows. 

Oklahoma’s standards and assessments and ACT’s Standards for Transition have 
significant strengths: 

• Oklahoma’s standards in English language arts and mathematics are comprehensive, 
written in clear, jargon-free prose and typically expressed in measurable terms. 

• Oklahoma’s standards are generally compatible with ACT’s Standards for Transition, 
which focus on essential content and skills necessary for college admission. 

• Oklahoma’s Core Ciuriculum Tests generally align well with those standards identified 
as appropriate for testing at the state level. 

• Oklahoma’s standards in mathematics lay a strong conceptual foundation in grades K— 5 
and, overall, contain a thoughtful, well-developed sequence of knowledge and skills. 

• Oklahoma has made the right decision by administering assessments that measure 
students’ ability to compose written prose. 

Despite these strengths, there are concerns about shortcomings in the standards and 
assessments that Oklahoma should address: 

• Oklahoma’s standards sometimes lack specificity, and this lack of precision clouds the 
level of rigor expected. Clarifying the level of rigor is critical and could be accomplished 
by using suggested reading lists in English language arts and including sample tasks in 
mathematics, for example. 

• Oklahoma’s standards in English language arts lack a clear and consistent progression of 
knowledge and skills from grade level to grade level. Making the progression explicit and 
ensuring the most important content receives the most emphasis would have multiple 
benefits — it would present a more coherent picture of the subject matter, add greater 
precision to the level of rigor the state expects, and sharpen the focus and organizational 
structure of the standards. 
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• Oklahoma’s standards do not clearly articulate how students’ writing skills are expected 
to develop over time and should be revised to specify the qualities the state expects 
students to demonstrate at each grade level. 

• Mathematics standards, while generally comprehensive, treat certain topics inadequately, 
particularly in algebra and in probability, statistics and data analysis. In revising its 
standards, Oklahoma should consider fortifying these topics, as today’s workplace 
demands a level of analytical and statistical skill not required in earlier generations. 

• The level of rigor on the Core Curriculum Tests fluctuates. In English language arts, it is 
appropriate for grade 5, too low for grade 8, and nearly at the appropriate level on the 
English II test. In math, tests in grades 5 and 8 sometimes measure standards from earlier 
grades and emphasize recall and procedural knowledge at the expense of more 
intellectually demanding skills. The level of rigor of the Algebra I test is more on target. 
As the state fully develops its grades 3-8 system of tests aligned to the standards and 
vertically aligned to each other, it should even out the level of rigor so a tight fit results 
between what the standards call for and what the Core Curriculum Tests measure. 

• Although ACT standards and Oklahoma’s standards are generally compatible, ACT tests 
in English, reading and math are not strongly aligned to Oklahoma’s standards. The gap 
can be narrowed by judiciously augmenting the state’s system of standards and tests with 
key content from the ACT. 
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Major Findings: Assessment System 



Oklahoma’s assessment system includes a mix of tests at various grade levels: a norm-referenced 
test in reading and mathematics at grade 3; criterion-referenced tests in several subjects at grades 
5 and 8; and an end-of-instruction test of high school English, with an algebra test coming soon. 
In addition, nearly all districts administer ACT’s EXPLORE and PLAN in grades 8 and 10 and a 
substantial proportion of students takes the ACT in grades 1 1 and 12. This array of assessments 
has certain strengths. 

Strengths of the Assessment System 

It provides measures of student progress against the state standards, as well as 
measures that permit comparisons with students in other states. 

Oklahomans deserve to know if students are meeting the standards the state has set; Oklahoma 
Core Curriculum Tests and end-of-instruction tests are designed to provide that information. 

Such tests also encourage teachers to focus on the standards and to design instructional programs 
that will enable students to reach them. 

At the same time, the state also administers a norm-referenced test — currently the Stanford 
Achievement Test, 9'*’ edition (SAT-9), which shows how students perform relative to other 
students nationally. These tests provide some perspective on student performance, much as 
growth charts show whether children are gaining weight and growing in height at the same rate 
as other children typically do. The state also participates in the National Assessment of 
Educational Progress (NAEP), which provides comparative information about state-level 
performance. 

Many states opt for one type of test or the other. In those cases, some information is missing. 
While comparisons also help students, parents and educators understand student performance, in 
Achieve’s view, the measure of performance against standards is critical. 

It includes tests in a broad number of subjects. 

Oklahoma’s decision to test students in history, government, geography, science and the arts 
commendably places these subjects on a par with reading and math and ensures that they are not 
neglected in the curriculum. This wide array of state tests makes Oklahoma rare among the 
states. Although nearly all states test students in reading and math, 29 states use science tests 
aligned with standards, and 22 states use aligned history/social studies tests. Only six states test 
students in the arts. 

Achieve’s concern, however, is that loading all of the tests into two grade levels places a 
tremendous burden on 5*- and 8**’-grade teachers. Although tests at different grade levels can 
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measure accumulated achievement, these teachers face the lion’s share of the responsibility and 
the time commitment for preparing students for tests in a broad range of subjects. 

The system uses the ACT to good advantage. 

Unlike other states, where college admissions tests assume great importance in students’ lives 
but are separate from schools’ efforts to raise achievement, Oklahoma has deliberately set out to 
make the ACT an integral part of its effort to improve the academic preparation of middle and 
high school students. Through the initiative of Chancellor Hans Brisch and the State Regents for 
Higher Education, the state has steadily increased the proportion of 1 1‘*’ and 12‘*’ graders who 
take the ACT and has raised average scores at the same time. Some 7 1 percent of the class of 
2000 took the exam, compared with 64 percent of the class of 1994. Oklahoma’s average ACT 
score rose faster than the national average during the 1990s. This is commendable; typically, as 
more students take such a test, the average score decreases to reflect the participation of students 
who previously may not have intended to go to college or prepared themselves for college. 

In addition, the state has helped students prepare for the ACT and for their lives after high school 
by implementing preparatory programs in earlier grades. Under those programs, virtually all 
schools in the state administer the EXPLORE exam in 8**’ grade and the PLAN exam in 10* 
grade. Both tests are linked to the ACT and provide students and teachers with information about 
students’ strengths and weaknesses in advance of the admissions test. Results are encouraging; 
the percentage of Oklahoma’s first-time college students enrolled in remedial courses has 
declined, and minority students have made significant gains on the ACT. 

Areas for Improvement 

Both Oklahoma and ACT assessments have many worthwhile features. But despite these 
strengths, the assessment system raises a number of questions and poses some concerns. 

The array of tests is disjointed and may send mixed signals to students and schools. 

As a test designed to be used in many states, the 3'^‘*-grade norm-referenced test was not designed 
to measure Oklahoma PASS standards. As a result, teachers may choose to focus more on the 
material on the test than on the standards, thus creating mixed messages about what is most 
important to learn. Similarly, high school teachers face the predicament of preparing students to 
take the ACT and are thus obliged to consider reinforcing skills not specified in PASS. 

The array of tests may also cause confusion because they report different types of results in 
different ways. The results of norm-referenced tests show percentile rankings, or the 
performance of Oklahoma students compared with a national sample. For example, in 1999, 
Oklahoma 3'^'* graders performed in the 66* percentile, meaning that they performed better than 
66 percent of the nation. The results of the criterion-referenced tests, however, tell another story. 
They show the proportion of students who perform at the unsatisfactory, limited knowledge. 
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satisfactory and advanced levels. In 2000, for example, 61 percent of Oklahoma S'** graders 
performed at the satisfactory level, and 15 percent performed at the advanced level in reading. 

Without a way to relate satisfactory performance on the Core Curriculum Tests to percentile 
rankings — which may not be possible because the tests measure different things — it is not 
possible to compare the two sets of results. As a consequence, parents, educators and the public 
learn one thing about 3'^‘*-grade performance and another thing about 5*-grade performance, and 
there is no way to tell whether one is better than another. Moreover, the two sets of results may 
conflict. For example, norm-referenced tests showed that performance in reading improved 
between 1998 and 1999, while Core Curriculum Test results showed performance stable over 
that time. What message do parents and the public receive about the state of student performance 
in Oklahoma? 

The disjointed nature of the assessments is particularly acute at the high school level. There, all 
students take the end-of-instruction tests, and about two-thirds of students take the ACT. Yet, as 
the Achieve analysis shows, a substantial portion of the content of state standards is not assessed 
by ACT English and mathematics tests. Preparing students for both sets of tests poses a 
challenge for teachers. 

In addition, the high school test sequence does not correspond to the course requirements in 
mathematics. For example, although students must take at least three years of mathematics (and 
perhaps four, if the legislature adopts Superintendent Sandy Garrett’s proposal to require four 
years of mathematics to graduate), they are tested only in algebra, which for many students is the 
first or second course in the sequence. Without an examination to measure attainment of a certain 
body of content, how can the state ensure that students continue to take academic coursework? 
The lack of additional tests in mathematics is mitigated in part by the fact that over 95 percent of 
the state’s 8* and 10**' graders are tested with PLAN and over 70 percent are tested with the 
ACT. However, having state end-of-course tests would fortify Oklahoma’s efforts to upgrade 
mathematics education. 

There is an issue for students enrolled in Oklahoma’s CareerTech programs. The Achieve review 
team noted that these programs receive a substantial share of state education funds and that they 
have strong support from rural commimities and their representatives in the legislature. The team 
believes that such students should not be exempted from the same standards all students are 
expected to demonstrate. By allowing CareerTech students to take courses that may not match 
the academic rigor of the courses taken by other students, or by creating a separate CareerTech 
system that would pull such students out of the academic requirements altogether, the state 
would be creating a two-tiered system of education — precisely the outcome a common set of 
standards for all students is intended to eliminate. 

To their great credit. Superintendent Garrett and the state department leadership have been 
resolute in not allowing watered-down CareerTech courses to supplant the academic 
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requirements. The department needs to continue to be vigilant to ensure that this practice 
continues. 

Recommendations for Building a Stronger Assessment System 

As Oklahoma moves forward to improve its system of assessments, Achieve recommends that 
the state commit to the following goals: 

Develop a coherent set of tests in grades 3-8. 

While the current system provides a great deal of information on student and school 
performance, testing in grades 3-8 will fill in the gaps in the current system that make it difficult 
to track student progress over time. As Texas has found, the ability to monitor performance from 
year to year has been extremely valuable to teachers and’ parents; it is a major reason the new 
federal law requires annual testing by every state. 

Testing in every grade also could reduce some of the uneven weight on teachers in the few 
currently tested grades by allowing the state to spread some subjects out over two or three years. 
For example, the state might consider moving its grade 5 science test to grade 6, while 
maintaining, and perhaps combining, history, government and geography in grades 4 and 7. 
Testing science at grade 8 will keep the state in alignment with the NAEP testing schedule. 

In developing the additional tests, Oklahoma will want to make sure that the overall system 
provides coherent information from year to year. The norm-referenced test in grade 3 might 
provide some important information, but it does not indicate whether students are meeting 
standards for that grade, and it is virtually impossible to track progress from performance on a 
norm-referenced test to performance on a criterion-referenced test in grade 4. The goal is to have 
a set of tests that supply coherent information regarding student performance against state 
standards over time. It thus makes sense for Oklahoma to have a set of grades 3-8 tests 
deliberately designed to measure performance against the state’s standards so essential 
information can be tracked from year to year. Maintaining a norm-referenced test not designed to 
match state standards as the sole measure of performance should not be a part of the assessment 
system. 

This does not mean that Oklahoma necessarily needs to develop the equivalent of its current 
grades 5 and 8 tests for grades 3, 4, 6 and 7. There are several ways the state can accomplish the 
goal of providing coherence without producing whole new tests. One way is to develop a hybrid 
test that combines a lower-cost commercially available test with items designed to measure state 
standards. California has taken this approach. There, the state initially administered a norm- 
referenced test, SAT-9 — the same test Oklahoma currently uses in grade 3. But the state 
immediately worked with the test’s publisher to augment SAT-9 with additional test items 
created to measure standards not tapped by SAT-9. Over time, the proportion of SAT-9 items on 
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California tests has declined, while the proportion of standards-referenced items has increased, 
moving California to a wholly standards-based test. 

Similarly, Delaware’s testing program embeds an abbreviated version of SAT-9 within a state 
test that was designed to measure performance against the state’s standards. This arrangement 
offers a number of advantages. It provides parents and schools with information on progress 
against the standards and comparisons with the nation as a whole; it also allows the state to 
equate results from year to year. 

Significantly, the Oklahoma Education Association proposed developing a hybrid test during the 
2001 legislative session. The association called the proposed test an “enhanced” norm-referenced 
test. Using hybrid tests in every grade would cost less than developing completely new tests for 
every grade, while at the same time providing coherent information from grade to grade. 

Another option would be to leave the current tests in place while adding shorter versions of the 
tests in intervening grades. These shorter versions might consist of fewer questions and would 
take less time to administer, but they would provide a snapshot of student performance in reading 
and mathematics that would enable parents and educators to gauge student progress over time. 
The detailed information about performance that the 5'*’' and 8‘'’-grade tests provide allows 
schools to receive enough information to adjust their instructional programs. 

Oklahoma also could collaborate with other states in developing common tests based on similar 
standards. An example of such a collaboration is Achieve’s Mathematics Achievement 
Partnership (MAP) — an effort involving 14 states meant to redefine mathematics education in 
the middle grades by using international benchmarks to build professional development, teaching 
tools and a common 8‘'’-grade test. 

Develop high school tests that measure what students are expected to learn. 

Oklahoma has strengthened requirements for high school course-taking and has developed end- 
of-instruction tests in English II, Writing (English II), Algebra I, Biology I and U.S! History 
designed to measure what the state expects students to learn in these key subjects. But there is a 
mismatch between the course requirements and the test requirements. The tests measure only 
coursework that could be completed by the end of grade 10. 

Parents and the public have no assurance that students actually are learning upper-level material 
— even if students enroll in upper-level courses. The danger is that schools could label a course 
“Algebra II,” but substantially dilute the content. Reviewers were told this was a real possibility 
in that currently no strict guidelines for making entries on the student transcript are in place. 

The only way to ensure that students are learning what they are expected to learn is to establish a 
common core that reflects what all students are expected to know and be able to do by the time 
they graduate and develop tests to match. 
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One way to do this would be to augment the system of end-of-course tests to include measures of 
the higher-level coursework students are expected to master. Maryland, for example, has 
developed math tests in algebra and data analysis that students, starting with the class of 2007, 
will be required to pass to graduate; the state also is developing a test in geometry. 

Another approach, which a number of states are employing, is to create a single high school 
exam that includes content from the common core. If Oklahoma chooses to go this route, the 
state should build on its existing system. Right now, the ACT has widespread credibility 
throughout Oklahoma, and over 70 percent of the state’s graduates take the exam. A new set of 
high school exams could build on the ACT much in the way new tests in grades 3-8 can build on 
existing commercially available tests. By examining the content of the ACT and developing 
items that fill in the gaps between the ACT and state standards — for example, by adding an 
assessment of students’ written prose, as the state is already planning to do — Oklahoma can put 
in place a test for all students that provides information considered credible without the expense 
and duplicative effort of developing a parallel set of tests. Colorado and Illinois are currently 
pursuing variations on this approach by requiring all students to take the ACT as part of their 
assessment batteries. 
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Major Findings: Accountability System 



In addition to implementing an extensive testing system, Oklahoma also has tried to make the 
test results count in a variety of ways by holding schools and students accountable for results. 
Oklahoma has attached consequences to performance to create incentives for improvement. It 
publicly reports results. It labels schools as “low performing” or “high challenge.” It links the 
awarding of driver’s licenses to test results. 

Strengths of the Accountability System 

Reports on performance are widely disseminated. 

As intended by the legislature, the Office of Accountability provides independent reports on the 
condition of education in Oklahoma. Its reports are distributed to every school, and principals are 
required to distribute them to parents as well. The news media regularly use the reports to 
provide information about schools and school system performance. As noted earlier, the State 
Department of Education also produces annual reports on performance. These are distributed 
throughout the state as well. 

High stakes appear to motivate students and schools to succeed. 

In the few instances in which Oklahoma has tied significant consequences to performance, these 
actions appear to produce effects. The most visible is the requirement that students pass the 8‘^- 
grade reading test to earn a driver’s license. This requirement is well known throughout the state, 
and the fact that students take it seriously undoubtedly contributes to their high performance. The 
proportion of 8‘^ graders scoring satisfactory or above on the reading test climbed from 70 
percent in 1996 to 81 percent in 1999. 

Similarly, the goal that 90 percent of graders will be at grade level in reading has prompted 
significant changes. Under the Reading Sufficiency Act, school districts are at least partially 
reimbursed by the state for providing specialized tutoring for primary students in need of reading 
remediation. Third graders also may be recommended for retention, with the parent or guardian 
being involved in the decisionmaking process. Setting the goal also has sparked a tremendous 
demand for professional development. 

Higher education is a genuine partner in school accountability. 

Unlike other states, where colleges and universities have tended to blame the schools for failing 
to produce qualified applicants, higher education in Oklahoma has worked directly with the 
schools to improve preparation and to make sure that the preparation pays off in admission and 
placement. Such efforts help build support for graduation requirements and tests by showing 
students and parents that performance opens doors. 
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In Oklahoma, the regents have used the tools at their disposal to make performance pay off. They 
have raised the coursework requirements for admission, encouraged widespread use of the ACT 
and set standards on the exam for placement. By taking full advantage of the Educational 
Planning and Assessment System, the regents are able to track student performance 
longitudinally. In addition, the legislature also has authorized a scholarship program based on 
coursework and ACT scores. 

Areas for Improvement 

Despite these strengths, the accountability system raises a number of questions and poses some 
concerns. 

It is unclear how useful test results are to schools and the public. 

To their credit, the state and its testing contractors produce a number of reports to make test 
results public. The test contractor produces reports to parents on individual children’s test scores. 
The State Department of Education produces an annual volume. Investing in Oklahoma, which 
reports statewide performance on a range of indicators, including state tests, the ACT and NAEP. 
The department also produces annual report cards for each school and district that provide 
information on test results and other performance indicators. The Office of Accountability also 
provides different school, district and state report cards that show test scores, other performance 
measures and contextual data, such as student demographics. The reports also compare school 
performance with that of the district and state. 

While there is an ample supply of information, it is not clear it is delivered in the most effective 
manner. Oklahoma citizens are in danger of being information rich and insight poor. For one 
thing, the reports from the State Department of Education and the Office of Accountability are 
potentially confusing. They use different data — the Office of Accountability reports lag behind 
the department’s by a year because they are based on data from the department — and the 
agencies often put a different “spin” on the results. For example, the Office of Accountability 
reports only the percentage of students who perform at the satisfactory level and above, while the 
department has shifted to describing four levels of test performance — unsatisfactory, limited 
knowledge, satisfactory and advanced. It seems it would make sense to have a single report that 
draws from the strengths of each, or at the very least two reports that dovetail, working in 
conjunction with one another to send a single, focused message. 

Our review team also heard differing opinions about reports to schools. Some schools seemed to 
find the reports helpful for planning improvements, while others did not. At least one district, in 
fact, developed its own tests to provide the diagnostic information officials felt the state tests 
lacked. The district tests, administered in grades 3, 4, 5, 8, and Algebra 1 and Biology, are 
developed by teachers to align to state standards. They report student performance on each of the 
state’s objectives tested and provide results more quickly than state tests do. 
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The state has not done enough to ensure that all students, particularly traditionally low- 
performing minority groups, are achieving high standards and are being assessed. 

Like many states, Oklahoma has a substantial minority population. Students from minority 
groups tend to lag behind white students in performance. Representatives from several minority 
grodps expressed grave concerns about the lack of attention to these achievement gaps. A 
particular concern was that, despite a federal requirement to break down test results to show 
performance by racial and ethnic groups, the state did not publish disaggregated results to be 
included in school and district reports until 2001. This was one factor the U.S. Department of 
Education cited in granting Oklahoma a waiver from the 2001 deadline for implementing a final 
assessment system under the 1994 federal education law that became No Child Left Behind. 

Some Latino parents said it was easy for English language learners to be exempt from state tests. 
Although state policy appears to require such students to take tests after a year in the United 
States, parents said students have been exempt for several years. With no information on their 
performance, their needs are unlikely to be addressed. 

Similarly, representatives of Native American organizations expressed serious concerns about 
the lack of attention to the performance of Native American students as well. Fortunately, this 
situation appears to be changing — and providing disaggregated data is the first step in 
improving the performance of minority populations. 

Current methods of rating school performance are not well understood and do not lend 
themselves to targeting resources effectively or to promoting continuous improvement. 

Like a number of states, Oklahoma rates schools by determining the proportion of students who 
reach the satisfactory level on state tests. Schools are designated as “low performing” if more 
than 30 percent of students score below the satisfactory level in both reading and math; they are 
designated as “high challenge” if they receive a low-performing rating for three consecutive 
years. 

This method appears to focus in on schools in the most need, but our reviewers heard a number 
of concerns about the approach. One concern is that the methodology used to set performance 
levels is not well understood and participation in the process is limited to educators. This has 
raised questions about the validity of the school ratings for some observers. 

In addition, the expectations seem to be quite low and easily achieved. Because the “low- 
performing” rating requires low scores in two subjects, a school can shed the designation by 
raising performance slightly in one subject one year (or simply benefit from a higher-performing 
5‘ or 8*'’-grade class). Few schools stay on the list long enough to earn the high-challenge 
designation. In fact, the list appears to be something of a revolving door; schools are designated 
as low performing one year, get off the list the next, then slip back and get on the list the 
following year. 
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This type of system does not allow the state to do what rating systems should allow it to do: 
namely, target resources to the schools that need them the most and encourage all schools to 
improve performance continually. 

In contrast, a number of states have set improvement goals for each school. Some use multiple 
levels of school performance analogous to the multiple levels of student performance Oklahoma 
uses. The approach provides goals for all schools, even those meeting standards, to strive toward. 
For example, if a school were rated “satisfactory,” indicating that it had met standards, it could 
still have a goal of reaching the “advanced” level. 

Other states set specific targets for improvement for each school. California requires each school 
to demonstrate a certain level of growth in achievement each year. The state provides financial 
rewards to schools that exceed their targets and provides resources and the possibility of 
intervention for schools that fail to meet them. The state also requires each racial and ethnic 
group of students within a school to meet growth targets. 

Oklahoma, to its credit, is shifting to a system like California’s, using an Academic Performance 
Index (API). Under the new system, each school and district would receive a rating based on 
state test results; school completion (attendance and graduation rates); and academic excellence 
(ACT scores. Advanced Placement course offerings and college remediation rates). Test results 
would be computed using an index that assigns weights to each level of performance on the state 
test, giving schools and districts higher scores for moving students to higher levels of 
performance. The overall result would be a score, with 1500 as the highest and 1000 as the state 
average. Each school and district would have to raise its API score each year by 5 percent of the 
difference between its API score and 1500. 

This system has the commendable asset of including a number of factors in school rating, rather 
than relying solely on test scores (although test scores, appropriately, are required to constitute 
the majority of the API). This is in keeping with the new federal law that permits the use of 
indicators other than test results in rating school performance, providing the use of these 
indicators does not reduce the number of schools that would be identified on the basis of test 
results alone. 

To ensure that the system serves the purpose of helping to guide school improvement, though, 
the state should publish not only the index, but also the data making up components of the index 
so that schools know where to target resources to meet their goals. 
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Interventions for low-performing schools need to be strengthened. 

While school ratings can by themselves be powerful means of encouraging schools to improve 
performance, accountability systems typically include methods of intervention in schools 
designated as low performing. Generally, such systems provide assistance, in the form of 
additional resources or technical assistance, to help schools turn themselves around. If such 
assistance does not result in improvement, states also have the authority to intervene in more 
drastic ways by replacing the staff, allowing children to transfer to another school or, in some 
cases, closing the school altogether. 

Oklahoma provides some resources and assistance to low-performing schools, and more for 
high-challenge schools. But it is not at all clear such efforts are effective. As noted previously, 
schools frequently fall in and out of low-performing status, so whatever assistance manages to 
help them in one year appears to provide short-term benefit at best. The state has a well-regarded 
system of professional development institutes, which could provide needed assistance to low- 
performing schools, yet there is no link between the accountability ratings and the ihstitutes. 

In addition, sanctions at the state’s disposal have not been used in powerful ways. To be sure, the 
state has intervened in one school, but that action is seen as an isolated incident. Schools around 
the state do not consider the possibility much of a threat. Oklahoma is far from alone in this 
regard. Nationwide, although 20 states have authority to impose sanctions on persistently poor- 
performing schools, only a handful of states have actually used such authority, and only in a 
smattering of schools. Yet when they have, the actions made a difference. In Maryland, for 
example, the state’s decision in 2000 to turn over the management of four low-performing 
schools to a private firm sent a strong signal throughout the state that poor performance would 
not be tolerated. 

The system has few stakes for anyone. 

Except for 8“’ graders and elementary school teachers and principals, there are no significant 
consequences for performance — good or bad — for anyone in the system; Students must take 
certain courses to graduate, but they can get by without learning any more than necessary to pass. 
Teachers and administrators can see their schools rated as low performing or high challenge, but 
not much happens to them if that is the case. Although the state has intervened in a few 
instances, the possibility of state action does not appear to be much of a motivation to educators. 
As a result, there is little incentive for improvement. 

Not surprisingly, then, performance does not appear to drive behavior the way it does in other 
states where the consequences are more significant. In Massachusetts, for example, a 
requirement that students pass state tests to graduate from high school has sparked massive 
efforts to redesign teaching and learning and substantial remedial help for students at risk of 
failure. These efforts have dramatically paid off. In 2001, the first year the tests counted as a 
graduation requirement, 82 percent of 10*^ graders passed the English language arts test, up from 
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72 percent on the test’s first administration in 1998; 75 percent passed the mathematics test, up 
from 48 percent three years earlier. 

Likewise, in Texas, where schools are accountable for student performance, teachers and 
administrators across the state remain focused like a laser beam on their students’ performance. 
Their work is driven by the need to raise performance. As a result, statewide results have risen 
steadily over time. The percentage of students passing the reading test rose from 74 percent in 
1994 to 89 percent in 2001, and the proportion passing the mathematics test rose from 58 percent 
to 90 percent over that period. 

Oklahoma’s system of career and technology education is not well aligned with 
secondary and postsecondary expectations, nor is it routinely subjected to academic and 
fiscal accountability. 

Reviewers heard repeated concerns from political and business leaders and educators that while 
Oklahoma had once been a model for the nation in terms of career education, it was no longer 
staying apace. One concern is the lack of a smooth fit between career education and the rest of 
the educational system, specifically the comprehensive high school program and college- 
entrance requirements. Leaders from various sectors expressed unease that, while the State 
Department of Education and the State Regents for Higher Education have been strengthening 
the academic preparation of high school students, students in the CareerTech sector appear 
headed on a different path that may not be as rigorous as the academic requirements for those in 
the general education system. As a result, CareerTech students may be facing limited 
opportunities should they choose to pursue postsecondary education. 

A second concern is the sense that many of the CareerTech programs are not preparing students 
for the current and future job market. While the programs may have been designed at a time 
when the industrial economy was dominant, the programs may be less relevant now in the 
information age. 

A third concern is the high cost of the CareerTech centers in a state strapped for financial 
resources. In part because of the way the CareerTech system is funded, the centers appear to 
consume a disproportionate share of the state education budget. This is particularly worrisome 
since the centers have not yet produced evidence of their effectiveness. 

Educators, parents and citizens are unable to track student performance over time. 

If students leave a school for another school in the same district or in another district, educators 
have no way of following the student to track his or her progress. Several states, such as Ohio, 
have developed extensive tracking systems that enable schools to monitor the progress of 
students over time without compromising privacy. Ohio’s system includes a wealth of data on 
student achievement, including test scores, grades and course-taking, as well as information on 
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the costs of instructional services. In that way, the system allows educators to track the 
management of districts and schools, as well as student progress. 

Texas has operated a similar system since 1995. The system also provides reports that show how 
students of various races and ethnicities and economic statuses perform. Oregon is making 
significant progress in implementing a secure student ID project to permit the State Department 
of Education to keep track of individual students’ test scores, evaluate programs like Title I and 
migrant services, and facilitate online testing of students. 

Recommendations for Building a Firm and Fair Accountability System 

As Oklahoma moves forward to improve its assessment and accountability systems. Achieve 
recommends that the state: 

Report results more clearly and make them more useful to schools and the public. 

Score reports are meant to serve several critical functions. Not only should they provide essential 
information for students, parents, teachers and schools, but the way in which the data are 
reported should make it straightforward for educators to pinpoint weaknesses in the instructional 
program to address them head on. Moreover, the linkages between school-level score reports and 
a state’s accountability system, including its API, should be spelled out. Oklahoma is not alone 
in struggling to construct reports that encourage stakeholders to take constructive action. In fact 
some states are beginning to call for enhancement of their current score reports in their latest 
requests for test publisher proposals. 

Large urban districts are equally concerned with this issue. The New York City Board of 
Education recently contracted with The Grow Network, a private firm based in New York, to 
produce a series of user-friendly, aligned reports for parents, teachers and principals that 
summarize achievement in English language arts and mathematics. The reports provide clear 
information on the extent to which students, classrooms and schools are meeting standards. More 
significantly, the reports also offer suggestions for how parents, teachers and principals can use 
the test data. For example, the parent report lists resources available in the local public library. 
The teacher report provides explanations of pertinent topics and offers effective teaching 
strategies, and the principal report allows administrators to look at student proficiency across 
classrooms and grades. 

Break down test results by race and ethnicity for every school. 

Until late 2001, Oklahoma’s reports on school and district performance showed how schools and 
systems performed overall without indicating how groups of students within each school and 
school system performed. In contrast, state reports provided information on different groups of 
students and showed significant gaps in performance between whites and ethnic minorities. 
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Breaking down results by race and ethnicity for individual schools can be extremely powerful. 
Schools that thought that they were doing well because their overall results were good found that 
they had rested on the performance of their majority students, while they did not effectively 
serve their minority students. This information led to significant changes in practice across the 
state. The new federal law will require every state to report annually such results for every 
school and school district. 

In providing the results by race and ethnicity, Oklahoma needs to take care to provide results for 
groups that are large enough to be statistically valid (providing information by gender, special- 
education status and English proficiency also is important and will be required by the new 
federal law). Where such information exists, it should be public. Only then will members of 
minority groups feel that they “count” in Oklahoma. 

Broaden the representation of participants in the standard-setting procedures for the 

state assessments. 

One of the critical elements in establishing performance standards or achievement levels is 
defining the representation of key advocates on the panels that identify the levels of achievement 
students should meet. Some states, such as Massachusetts, have attempted to set rigorous 
standards for their students with the participation of community representatives as well as 
experienced educators and have thus encouraged a more public participation and awareness of 
the state’s expectations. In addition to widening the participation in standard-setting, states are 
best served by having an independent technical advisory committee review the proposed 
performance standards prior to their formal adoption by the state. 

Sharpen the method of rating school performance. 

School ratings can serve three purposes. They identify the schools most in need of improvement. 
They provide information on school quality that is essential for accountability. They create 
incentives for all schools to raise performance levels. Oklahoma’s current system accomplishes 
the first of these, although not as well as it could. The shift to the API could help accomplish the 
other two purposes as well. 

However, by focusing only on performance gains, the new system fails to measure two other 
characteristics of school performance that are worth knowing: the school’s current level of 
performance and its success in reducing achievement gaps. Of course, a school that has improved 
substantially deserves recognition, and the growth targets enable all schools, even those that have 
the farthest to go to attain proficiency, to earn recognition. Yet since the goal is proficiency, 
schools that have reached or approached that goal also deserve recognition. So do schools that 
have narrowed achievement gaps. 
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Delaware is putting a system in place that rates schools on all three factors: the overall level of 
performance, growth over time and success in raising the performance of low-achieving students. 
Each school is compared to a statewide target. 

^ Provide direct assistance to low-performing schools and students. 

The Oklahoma State Department of Education has assigned teams to assist schools designated as 
low performing and high challenge, and some of these schools have received additional 
resources largely through federal programs designed to support school improvement. For 
example, 48 schools in 2000-01 received about $50,000 each under the Comprehensive School 
Reform Demonstration Program, which provides funds for schools to adopt a schoolwide, 
research-based design for improvement. 

Most of the assistance, though, went toward helping schools develop improvement plans. It is 
unclear whether the state provides any assistance in enabling the schools to carry out their plans 
or in making sure that they do — and that achievement rises as a result of the school’s actions. 

As several people told the Achieve review team, many schools raise performance enough in one 
subject to make their way off the low-performing school list, only to decline the next year and , 
get back on the list. 

Other states offer models for providing intensive assistance that appear effective in helping 
schools raise their level of performance. Some of the most effective strategies involve state- 
trained educators who provide intensive aid to troubled schools. For example. North Carolina 
sends trained “assistance teams” of experienced educators to help low-performing schools. These 
teams help schools write and implement improvement plans and recommend changes needed to 
bring about improvements. They also evaluate the principal and staff and can recommend 
replacing teachers or administrators. The program seems to be working: Of the 15 schools 
assigned assistance teams in the first year, 14 reached their target for improved performance. 
Thirteen schools improved enough to earn cash bonuses from the state. 

Some states employ regional centers to assist low-performing schools. In Florida, for example, 
the state’s regional school improvement teams work intensively with schools receiving a grade 
of “F” on the state’s rating system. The teams link schools with grant opportunities and other 
resources. 

Even as the state gears up to amplify interventions in failing schools, it must also consider 
interventions for failing students. It already has an excellent example in its early reading 
initiative, which was the right place to start and is likely to go a long way in helping to prevent 
student failure. However, in looking beyond this effort, reviewers did not see the kind of 
academic scaffolding in place that will be necessary if all students are to achieve proficiency in a 
rigorous, standards-based system.' In Massachusetts, by contrast, the state invested substantial 
resources to implement extra programs for students in danger of failing the graduation exams. 
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These programs, moreover, did not just give students more of the same; they tried new strategies 
to help students learn what they may not have mastered in the regular classroom. 

Apply real sanctions after continued failure. 

Intervention and assistance can help turn schools around. But what if schools do not improve? 
States must take more serious action. As governors, business leaders and educators at the 1999 
National Education Summit pledged in their Action Statement: “We will be prepared to 
restructure or reconstitute schools or provide parents and students with other options. The state 
has a solemn obligation to ensure that no child is trapped in a failing school.” 

Many states, like Oklahoma, have laws on the books to provide some sanction for schools that 
fail continually. But like most states, Oklahoma has used this authority sparingly — perhaps too 
sparingly. It also may be the case that imposed sanctions have not received sufficient publicity. 
Achieve’ s reviewers repeatedly heard that educators and the public do not appear to take the 
state’s authority to intervene seriously, and thus the authority is not the incentive it could be for 
failing schools to feel a sense of urgency in turning themselves around. In Maryland, by contrast, 
the state’s well-publicized action to turn four schools over to private management sent a clear 
signal throughout the state. 

Oklahoma may wish to think of expanding or developing more fully options for failing schools. 
One approach is to replace the management of persistently low-performing schools. Another 
option, \vhich Colorado will implement this year, allows districts to convert schools rated failing 
into charter schools. Still another approach, which the new federal law requires, would provide 
additional options for students in low-performing schools. Under the law, students can transfer to 
other public schools or receive supplemental services, like tutoring, from private providers. 

Schools are not the only institutions responsible for improving student achievement; districts and 
the state have important roles as well, and they all should be held accountable. In addition to its 
measures for schools, Maryland has been a leader in district accountability. In 1996, the state 
passed special legislation establishing unique governance arrangements and targeting programs 
and funds for the Baltimore City system, which serves about a third of the state’s minority 
student population and Title I participants, yet has the lowest wealth per pupil in the state. The 
state and the city instituted a new partnership, with a new school board and CEO as 
superintendent; in return the state substantially increased financial aid to the district. The 
achievement of Baltimore students seems to be on the rise — an encouraging sign. 

Create real incentives for students to achieve. 

Oklahoma has taken notable strides to make the high school diploma meaningful by 
strengthening graduation requirements and implementing tests that measure student achievement 
in designated high school courses. Such efforts surely have contributed to the state’s impressive 
record on the ACT — an increase in average scores at a time when participation increased. 
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However, these steps may not have gone far enough to create real incentives for students to 
achieve at high levels. Students who take the required coursework can earn as low as a “D” and 
still graduate. Even if students take the tests, their scores matter only if college admissions 
officials or employers look at their transcripts and factor in the results in their decisionmaking. 
There is no designated level of proficiency that students are expected to reach. 

Nearly half the states currently have in place, or will soon, a test that students must pass to 
graduate from high school. Several of these efforts have attracted controversy, but the Achieve 
team found real and widespread support for such a move in Oklahoma. Educators, business 
leaders and public officials recognize that a graduation test creates the proper incentives for 
students to study hard and succeed in high school. As the Massachusetts experience shows, when 
test performance counts, students rise to the occasion. 

This is not to say the system should be rigid. While elected representatives, higher education 
leaders and employers must unequivocally support the state’s system of assessment and 
accountability, they should also consider building a “safety net” for a limited number of students 
who may be able to demonstrate through other means that they have met state standards. For 
example, such students should have demonstrated good faith efforts to pass required assessments 
on multiple occasions, have come close to passing and have good grades in their academic 
coursework in a rigorous curriculum. 

^ Ensure that the career and technology education system works in conjunction with the 
K-16 system to prepare students for high-skill careers and continuing education. 

The issue of how best to prepare students for workplace success in a “knowledge economy” is 
taxirig many states. For more than a decade, the U.S. Department of Labor has consistently 
pointed out that low-skill jobs are on the wane and has urged that students be ready for 
continuing education even if they choose not to seek further education immediately after high 
school graduation. Current statistics from the UjS. Department of Education indicate that 75 
percent of high school students are choosing to enroll in some form of postsecondary education. 

It is clear to Achieve that Oklahoma should continue its push for a common core of high school 
courses, which terminate in the kind of end-of-instruction tests the State Department of 
Education already is putting into place and requiring for graduation. Adequate preparation in 
math is essential for higher education and the workplace, yet it is the area of study most in need 
of attention. Requiring four years of high school math is a solid approach, providing that 
substandard courses are not permitted to take the place of rigorous Algebra I, Geometry and 
Algebra II courses. Quality control is paramount. Another, perhaps interim, strategy is to require 
that students take a second year of math and pass a state test to graduate. 

Other states have managed to strengthen academic requirements for all students, including those 
in career and technical education. In 1997, New York established new graduation requirements, 
gradually replacing basic competency tests with the state’s more demanding Regents Exams in 
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English, mathematics, science and social studies (U.S. history and government and world 
history). To help support students in the transition, the traditional passing score of 65 was 
temporarily lowered to 55. These graduation requirements hold for all students, even those 
pursuing a technical program. New York has further worked out provisions for students to earn a 
technical endorsement on a Regents diploma or a Regents diploma with advanced designation. 
To make students eligible for these options, school districts must engage in a program approval 
process that culminates in a sign-off by the state. 

An Oklahoma version of stronger high school standards for all students should build on the 
significant partnership between the State Department of Education and the State Regents for 
Higher Education. The two sectors should work together, along with the business community, to 
define the knowledge and skills students need after high school and design tests that measure 
those abilities. But Oklahoma does not need to define those expectations on its own. Five states 
— Indiana, Kentucky, Massachusetts, Nevada and Texas — currently are working with Achieve 
and other national organizations in a partnership, known as the American Diploma Project, 
which is designed to help states analyze whether they have set the bar for graduation 
requirements in reading, writing and mathematics at the right level; stimulate demand — from 
higher education and employers — for standards-based high school assessment data in their 
admissions and hiring process; and create new high school graduation benchmarks in English 
language arts and mathematics that states can use to judge the quality and rigor of their standards 
and tests. In the spirit of the American Diploma Project’s goals, it seemed to reviewers that one 
innovation the regents would want to consider is instituting a uniform, common set of placement 
tests to be used by all institutions of higher learning in the state. 

In addition to strengthening the academic preparation of high school students in general and high 
school CareerTech programs in particular, Oklahoma also needs to ensure that the career and 
technical programs in particular succeed in their primary mission: placing students in entry-level 
jobs that have worthwhile career paths. 

Judging the ultimate success of the state’s system of career and technical education will require 
the state to track students beyond high school to determine if they found gainful employment in 
the field for which they were trained or if they successfully completed a program of continuing 
education or, ideally, both. 

Develop a student-tracking system. 

An expanded assessment system should be combined with a student tracking system that would 
enable parents and teachers to measure performance over time and determine the “value added” 
teachers and schools contribute to student achievement. The system also should follow students 
over time, even if they change schools. Such a system is particularly valuable in a state like 
Oklahoma where, the Achieve team heard, student mobility is quite high. 
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Some Oklahomans have objected to such a system on the grounds that it might invade children’s 
privacy and create a “Big Brother”-type database. But the experience in Ohio, Texas and other 
states with tracking systems shows that such fears may be unfounded. Georgia, in fact, has 
specifically prohibited the state from using Social Security numbers to track students and instead 
uses coded identification numbers that cannot be traced. 

Moreover, Oklahoma has experience with a student-tracking system in higher education. The 
regents are able to follow students as they move through colleges and universities to provide 
information to the high schools from which they graduated on their postsecondary performance. 
As the state looks into developing a tracking system for K-12, officials should work with the 
regents to build on their experience and make the systems compatible with, or linked to, one 
another. 
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Conclusion 



In making these recommendations to strengthen Oklahoma’s assessments and accountability 
system, Achieve is under no illusions that implementing them will be easy. Some of them will 
require funding, always a challenge in Oklahoma and particularly at a time when budgets are 
tight. Some of them will require more aggressive action by the State Department of Education, a 
step that goes against the strong tradition of local control over education in the state. And some 
will require a carefully thought out public awareness and engagement campaign. 

Despite these challenges. Achieve believes that the recommendations are feasible. What they 
require most of all is leadership. As we have seen in state after state, strong leaders can 
accomplish great goals — and, most importantly, produce great results in higher student 
achievement. 

Oklahoma has benefited from strong leadership in education. Political leaders have taken bold 
steps, beginning with the landmark reform law of 1990. The State Department of Education has 
led the way in developing standards and assessments and beginning a system of holding schools 
accountable for results. The State Regents for Higher Education have taken the initiative to 
strengthen students’ preparation for college. And the business community has shown that it is 
willing to organize and invest resources on behalf of improving education. 

The state’s leadership is going through a period of transition in 2002. The governor’s term is 
expiring. The state superintendent is running for re-election. And the chancellor of the board of 
regents is retiring. For these reasons, it makes sense for the business community — particularly 
through its institutional arm, OBEC — to take the lead in helping usher through the next stage of 
education reform in Oklahoma. 

In taking on this role, the business community can offer a number of advantages. Not part of any 
one sector of education, it can bridge K-12 and higher education and bring them together to 
build on the strengths of the initiatives they created. Business also is seen as a credible and 
independent voice throughout the state. And the success of the MAPS for Kids initiative in 
Oklahoma City in 2001 shows that business leaders can help achieve results. 

Perhaps the most critical initial task is to persuade Oklahomans that a stronger system of 
standards-based reform is essential. The leaders in government, education and business clearly 
know that it is, but many residents of the state may not be ready for major changes. By showing 
how the future of the state depends on a well-educated populace, business leaders and others can 
begin to develop the support needed to bring these bold reforms to fruition. 

The biggest strength of Oklahoma’s education system is the widely shared, genuine commitment 
to improving opportunities for all young people. By channeling this commitment into a single 
vision for a stronger assessment and accountability system, the state can make this goal a reality. 
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Appendix A: Review Team Biographies 



Ruben Carriedo 

Ruben Carriedo is a senior research associate at the University of Michigan School of Education. 
He is part of the Study of Instructional Improvement, a comprehensive longitudinal study of 
instructional interventions at 125 elementary schools across the coimtry. Carriedo has been a 
teacher, coach, counselor, and secondary school and central office administrator in New York 
City and San Diego public schools. He served as the assistant superintendent for the Planning 
Assessment, Accountability and Development Division in the San Diego City Schools from 1987 
to 1999. 

He has served on the California Assessment Program Policy and Technical Advisory 
committees. New Standards Project Technical Committee, the National Advisory Panel for the 
Center on the Reorganization and Restructuring of Schools, the National Advisory Panel on 
Chicago School Restructuring, PACE/Rockefeller Project Advisory Committee, Editorial Board 
of the Educational Evaluation and Policy Analysis Journal, and Advisory Committee of the 
Learning Research and Development Center at the University of Pittsburgh. He currently serves 
on the TIMSS-R Technical Review Panel, Advisory Committee on Education Statistics at the 
National Center for Education Statistics, Trustee Committee on Research and Development of 
the College Board, and Title I Assessment Committee at the National Research Council and 
National Academy of Science. 

He has worked as a consultant to the Spencer and MacArthur foimdations and Booz-Allen on 
Chicago school reform. In addition, he has consulted with the Ford Foundation to plan a national 
symposium on assessment and equity. He also served as a member of the national evaluation 
panel for the Children Achieving reform initiative in Philadelphia. 

Carriedo received his doctorate from the Harvard Graduate School of Education. 

Matthew Gandal 

Matthew Gandal is executive vice president of Achieve, Inc. He manages the Washington, D.C., 
office and is responsible for overseeing Achieve’ s major initiatives. These include the 2001 and 
1999 National Education Summits and a series of follow-up activities Achieve has launched to 
help states address the Summit challenges; the Benchmarking Initiative, which helps states 
compare their standards, assessments and accoimtability policies with those of other states and 
nations; the Mathematics Achievement Partnership (MAP), which is designed to help states 
improve curriculum and instruction in middle school math and measure student achievement 
using a common, internationally benchmarked 8*^-grade test; and the American Diploma Project. 

Before joining Achieve, Gandal was assistant director for educational issues at the American 
Federation of Teachers (AFT). He helped AFT launch a variety of programs and publications 
designed to support standards-based reform efforts in states and school districts. He was the 
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author and chief architect of Making Standards Matter, an annual AFT report evaluating the 
quality of the academic standards, assessments and accountability policies in the 50 states. He 
also authored a series of reports that compared student standards and achievement in the United 
States with those of other industrialized nations. 

Gandal, a graduate of the Maryland public school system, earned a bachelor’s degree in 
philosophy from Trinity College in Hartford, Conn. 

S. Paul Reville 

S. Paul Reville is a lecturer on education and coordinator of state relations at the Harvard 
Graduate School of Education. He also is the executive director of the Pew Forum on Standards- 
Based Reform. The forum conducts national policy seminars on key reform issues and provides 
advice and assistance to a handful of leading-edge states and urban districts engaged iii 
implementing systemic reform programs. Reville is the chairman of the Massachusetts Education 
Reform Review Commission. Appointed by the governor, the commission is charged with 
overseeing the state’s implementation of the historic Education Reform Act of 1993. 

Prior to his appointment at Harvard, he was the executive director and co-founder of the Alliance 
for Education, a privately supported, multiservice education foundation dedicated to improving 
public elementary and secondary education in Worcester and central Massachusetts. Reville was 
also the co-founder and executive director of the Massachusetts Business Alliance for Education 
(MBAE), an organization that provided key conceptual and political support for the 
Massachusetts Education Reform Act of 1993. He currently serves as MBAE’s executive 
director. From 1991 to 1996, he served a five-year term on the Massachusetts State Board of 
Education where he served on the executive cornmittee and chaired the Massachusetts 
Commission on Time and Learning. He has been a teacher and a principal in various schools and 
alternative programs. Reville is a trustee of Wheelock College and the Public Education Network 
and serves on various state and civic organizations and commissions. He has received numerous 
professional and civic awards and is a frequent speaker, writer and editor on educational matters. 

He received a bachelor’s degree from Colorado College and a master’s degree from Stanford 
University. 

Robert Rothman 

Robert Rothman was a senior project associate of Achieve from May 2000 through March 2002, 
where he developed and wrote a series of publications for the Public Leadership Initiative. 

Before joining Achieve, Rothman was a study director for the Board on Testing and Assessment 
at the National Research Council. He directed a study on assessment and teacher quality, which 
produced the report Tests and Teaching Quality: Interim Report. He also directed a study on 
standards-based assessment and accountability, which produced the report Testing, Teaching, 
and Learning: A Guide for States and School Districts. Previously, Rothman was the director of 
special projects for the National Center on Education and the Economy and a reporter and editor 
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for Education Week. He is the author of Measuring Up: Standards, Assessment and School 
Reform (Jossey-Bass Publishers Inc., 1995) and numerous book chapters and articles on testing 
and education reform. 

Rothman received his bachelor’s degree from Yale University. 

Jean Slattery 

Jean Slattery has been a consultant for Achieve since 1999 and currently serves as director for 
the Benchmarking Initiative. She was supervising director of curriculum development and 
support in Rochester, N.Y., from 1989 to 1997, with responsibility for overseeing the work of all 
subject-area directors in the K-12 instructional program. Her earlier responsibilities as a district- 
level administrator included serving as director of the middle school (1987-89) and junior high 
(1985-87) programs. During this period, she initiated Teachers as Partners, a peer-coaching staff 
development program funded by the Ford and Matsushita (Panasonic) Foundations. 

Slattery also is a peer consultant on standards and assessment for the U.S. Department of 
Education. She has served as a consultant to the Washington, D.C., school district; San Diego 
Unified School District; a Washington state consortium of rural schools; and the Alabama and 
Illinois Departments of Education. Slattery also has worked for the Council for Basic Education 
on projects involving the Flint Community School District, the Nevada Education Department 
and the Cleveland Municipal School District. 

Slattery received a bachelor’s degree in chemistry from Albertus Magnus College, a master’s 
degree in science education from Yale University and a doctorate in science curriculum from the 
University of Rochester. 

Janis Somerville 

Janis Somerville is the senior associate with the University System of Maryland and the National 
Association of System Heads (NASH), the association of CEOs of the 53 state systems of public 
higher education. She directs the NASH/Education Trust K-16 Network, through which those 
two organizations have come together to support a network of higher education, K-12 and civic 
leaders who are implementing statewide K-16 improvement efforts. Members of the network are 
committed to working both within their states and with their counterparts in other states to 
develop coordinated improvement efforts, kindergarten through college. These efforts have a 
single goal: improving student achievement at all levels and closing historic achievement gaps 
between students of different racial and economic backgrounds. 
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A native Pennsylvanian, Somerville began her work in education as a high school teacher in 
Trenton, N.J. She also worked for several years in higher education, including appointments as 
the senior academic officer for undergraduate education at Temple University and at the 
University of Pennsylvania. In 1989, she founded the Philadelphia Schools Collaborative, a joint 
venture of area foundations and the Philadelphia School District to bring K-,12, college and 
comrhunity resources together in a comprehensive high school restructuring and college access 
initiative. 

In 1994, she moved to Maryland to develop Maryland’s statewide Partnership for Teaching and 
Learning K-16; and in 1997, with support from The Pew Charitable Trusts, she came to NASH 
to develop the NASH/Educatioh Trust K-16 Network. 

Somerville received her bachelor’s degree from the Pennsylvania State University and her 
master’s degree in business administration from Harvard Business School. 

John H. Stevens 

John H. Stevens is the executive director of the Texas Business and Education Coalition (TBEC). 
TBEC is a statewide coalition of business executives and education leaders that works to 
improve the performance of the Texas public school system in terms of student achievement. 

The coalition seeks to influence the direction of state policy and encourages and supports the 
work of local business-education coalitions working to reform schools at the community level. 
Stevens has written numerous booklets, policy position papers and articles on educational issues 
and frequently speaks about the coalition’s reform agenda at education and business conferences. 

He represents TBEC on several committees, including The U.S. Chamber of Commerce 
Education, Employment and Training Committee; the Commissioner’s Advisory Committee on 
Accountability; the Advisory Committee for the Educational Productivity Council at the 
University of Texas; the Texas Society of CPAs Helping Schools Committee; the Education 
Committee of the Greater Austin Chamber of Commerce; the Advisory Committee for the 
Region 13 Cohort Administrator Certification Program; the Executive Advisory Board of the 
New Century Educational Leadership Program at Tarleton State University; and others. 

He currently is chairman of the new Texas Principals Leadership Initiative and vice chairman of 
the Advisory Board of the Texas Statewide Systemic Initiative for Science and Mathematics 
Reform. He also is a member of the English language arts writing team for the Texas Essential 
Knowledge and Skills clarification project. 

He was chief writer of the Basic Workplace Skills Training Manual for the American Society of 
Training and Development. Previously, Stevens created and managed a national training 
program, which involved over 200 events a year. His department provided support for education 
reform projects in school districts across the country. He conducted training programs for 
education leaders in Hungary, the Philippines, Poland, South Africa and other nations. 
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He created and served as executive producer of the award-winning television series, “Inside 
Your Schools” hosted by Steve Allen. The series focused on major issues in public education and 
appeared on more than 130 PBS stations and The Learning Channel. While working in 
Wisconsin from 1970 to 1975, he served on the State Superintendent’s Task Force on 
Educational Goals, the Advisory Committee to the State Legislature on Compulsory School 
Attendance and other committees. 

Stevens received his bachelor’s degree from Bemidji State University and did graduate work in 
psychology and school counseling at St. Cloud State University. 
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Appendix C: Interviews 



Keith Ballard 

Executive Director 

Oklahoma School Board Association 

Peter Beaver 

Oklahoma State Department of Education 
Kim Bishop 

House of Representatives 
State of Oklahoma 

Bill Bleakley 
Journalist 
The Gazette 

Kevin Boettger 
Superintendent 
Chandler Public Schools 

Bill Bray 

Oklahoma Rural Systemic Initiative 

Hans Brisch 
Chancellor 

Oklahoma State Regents for 
Higher Education 

Susan Broadbent 
Senate Staff 
State of Oklahoma 

Robert Buswell 
Executive Director 
Office of Accountability 
State of Oklahoma 



Dottie Caldwell 

Director of Professional Development 
Oklahoma Commission for Teacher 
Preparation 

Carol Cawyer 

Director of Program Accreditation 
Oklahoma Commission for Teacher 
Preparation 

Tinny Chang 
President 

Oklahoma Chinese Times 

Peter Churchwell 
President 

American Electrical Power 

Judith Coe 
Dean 

College of Education 
University of Central Oklahoma 

Glenn Coffee 
Senator 

State of Oklahoma 

Floyd Coppedge 
Secretary of Education 
State of Oklahoma 

Carolyn Crowder 
President 

Oklahoma Education Association 

Susan Crowder 
Principal 

Damaby Elementary School 
Union Schools 
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Bums Hargis 
Vice Chairman 
Bank of Oklahoma 

Matt Hesser 
Assistant Director 
Office of Accountability 
Education Oversight Board 
State of Oklahoma 



Ernie Diamond 
Education Research Specialist 
Office of the Secretary of Education 
State of Oklahoma 

Kathy Dodd 

Director of Testing and Assessment 
Union Public Schools 

Ray Doonkeen 
Chairman 

Hispanic Chamber of Commerce 



Drew Dugan 

Manager of Governmental Relations 
Oklahoma City Chamber of Commerce 

Katie Dunlap 

Assistant State Superintendent 
Oklahoma State Department of Education 

Brandon Dutcher 

Executive Director 

Oklahoma Council on Public Affairs 

David Du Vail 

Executive Director 

Oklahoma Education Association 

Charles R. Ford 
Senator 

State of Oklahoma 
Sandy Garrett 

State Superintendent of Public Instmction 
Oklahoma State Department of Education 

Ted Gillispie 
Executive Director 
Oklahoma Commission for Teacher 
Preparation 



Barbara Hoberock 

Journalist 

The Tulsa World 

Cal Hobson 
Senator 

State of Oklahoma 

Lori Howell 
Teacher and Sponsor 
Middleberg Public Schools 

Pat Howell 
Director 

Organization of Rural Elementary Schools 
Jerry Hsieh 

Office of Accountability 
State of Oklahoma 

Phyllis Hudecki 
Executive Director 

Oklahoma Business and Education Coalition 

Carlos Johnson 
Board Member 

Oklahoma State Chamber of Commerce 

Kathryn Jones 

Executive Director 

Higher Education Alumni Council 
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Theresa Kiger 
Teacher 

Clark Elementary School 
Union Public Schools 

Cindy Koss 
Team Leader 

Curriculum and Effective Schools 
Oklahoma State Department of Education 

Renee Launey-Rodolf 
Director of Assessments 
Oklahoma Commission for Teacher 
Preparation 

Sharon Lease 

Chief Academic Officer 

Oklahoma City Public Schools 

Adriana Lopez 

Parent and School Volunteer 

The Hispanic Chamber of Commerce 

Oklahoma City 

Ann Candler Lotven 
Dean 

College of Education 
Oklahoma State University 

Stan Lybarger 

.President and Chief Executive Officer 

Bank of Oklahoma 

Chaimian 

Oklahoma Business and Education Coalition 

Betsey Mabry 
Einstein Fellow 
Teacher of the Year 
Waller Junior High School 
Enid Schools 



Kevin McKinley 
Superintendent 
Bray-Doyle, Public Schools 

Pat McQuigan 
Journalist 

The Daily Oklahoman 

Dolores Mize 
Assistant Vice Chancellor 
Oklahoma State Regents for Higher 
Education 

Jennifer Morris 
Executive Director 

Academic Performance Index Research 
Oklahoma Department of Education 

Bill Nations 
Representative 
House of Representatives 
State of Oklahoma 

Sacra Nicholas 
Member 

Coalition for the Advancment of Science 
and Math in Oklahoma 

Dana Owens 

Assistant Professor 

University of Central Oklahoma 

' Quang Pham 

Engineering Manager 
Department of Environmental Quality 
State of Oklahoma 



Randall Rabum 
Executive Director 

Cooperative Council for Oklahoma School 
Administrators 
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Linda Rendon 

Liaison for the Hispanic Department 
Oklahoma State Department of Education 

John Rex 
President 
American Fidelity 

Larry D. Roberts 
Representative 
House of Representatives 
State of Oklahoma 

Cindy Ross 

Director of Academic Affairs 
Oklahoma State Regents for 
Higher Education 

Kay Ruelle 

Team Leader of Student Assessment and 
Planning 

Oklahoma State Department of Education 

Claudia San Pedro 
Senate Staff 
State of Oklahoma 

Hossein Sarjeh-Payma 

Chairperson 

Asia Society 

Dean Schirf 

Vice President 

Government Relations 

Greater Oklahoma Chamber of Commerce 

Vasithy Sengdara 
Bilingual Resource Specialist 
Asian American Student Services 
Oklahoma City Public Schools 



Terri Silver 
Board Member 

Oklahoma City Public Schools 
Tim Tallchief 

Oklahoma State Department of Education 

Nancy Tan- 
Budget Analyst 
Office of State Finance 
State of Oklahoma 

Debbie Ter lip 

House of Representatives 

State of Oklahoma 

Stuart Tonemah 

Oklahoma Rural Systemic Initiative 

Ron Walker 
Principal 

Jackson Middle School 

Barbara Ware 
Chair 

Oklahoma Commission for Teacher 
Preparation 

Stacey Weinand 

Coordinator of State Regents’ Math 
Initiative 

Kathleen Wilcoxson 
Senator 

State of Oklahoma 

Beverly Woodrome 
Executive Assistant 

Oklahoma Business and Education Coalition 
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Sue Woodward 
President 

Tulsa Association of Classroom Teachers 

Kathleen Workman 

Director of Elementary Curriculum 

Union Public Schools 
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Board of Directors, Achieve, Inc. 

Co-Chairs 

John Engler, Governor 
State of Michigan 

Louis V. Gerstner Jr., Chairman 
IBM Corporation 

Co-Vice Chairs 

Philip M. Condit, Chairman and CEO 
The Boeing Company 

Gray Davis, Governor 
State of California 

Board Members 

Keith E. Bailey, Chairman 
Williams 

Roy E. Barnes, Governor 
State of Georgia 

Craig R. Barrett, CEO 
Intel Corporation 

Frank Keating, Governor 
State of Oklahoma 

Gary Locke, Governor 
State of Washington 

Edward B. Rust Jr., Chairman and CEO 
State Farm Insurance 

Arthur F. Ryan, Chairman and CEO 
Prudential 

Bob Taft, Governor 
State of Ohio 

President 

Robert B. Schwartz 
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